No-free-lunch theorems have shown that learning algorithms cannot be universally good. We show that no free funch exists for noise prediction as well. We show that when the noise is additive and the prior over target functions is uniform, a prior on the noise distribution cannot be updated, in the Bayesian sense, from any finite data set. We emphasize the importance of a prior over the target function in order to justify superior performance for learning systems

Magdon-Ismail, Malik

Caltech Authors

English

accepted

Amari,S.,Murata,N.,M¨ uller,K.,&Yang,H.(1997).Asymptoticstatisticaltheory of overtraining and cross validation.

Asymptotic statistical theory of overtraining and cross validation.

Bayesian regression ﬁlters and the issue of priors.

Elements of information theory.

Gaussian processes for Bayesian classiﬁ-cations via hybrid Monte Carlo. In

Gaussian processes for Bayesian classifications via hybrid Monte Carlo. In

Limits on learning machine accuracy imposed by data quality. In

Limits on learning machine accuracyimposedbydataquality.InG.Tesauro,D.S.Touretzky,&T.K.Leen (Eds.),Neuralinformationprocessingsystems,7(pp.239–246).Cambridge,MA:

Lippmann (Eds.), Advances in neural information procesing systems,

Monotonic networks. In

Monotonicity hints. In

No free lunch for cross validation.

No free lunch for early stopping.

No free lunch theorems for optimization.

The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems.

The existence of a priori distinctions between learning algorithms.

The jackknife and the bootstrap.

The lack of a priori distinctions between learning algorithms.

The mathematics of generalization.

The nature of statistical learning theory.

No Free Lunch for Noise Prediction